Implement StringNormalizer #69
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yuslepukhin
requested review from
RyanUnderhill,
pranavsharma,
wschin and
xadupre
yuslepukhin
commented
Nov 30, 2018
xadupre
reviewed
Nov 30, 2018
xadupre
reviewed
Dec 1, 2018
xadupre
reviewed
Dec 1, 2018
wschin
reviewed
Dec 1, 2018
xadupre
reviewed
Dec 1, 2018
xadupre
reviewed
Dec 1, 2018
yuslepukhin
force-pushed
the
dmitrism/string_normalizer
branch
from
19755fe to
b636ae7
Compare
RyanUnderhill
previously approved these changes
Dec 4, 2018
RyanUnderhill
previously approved these changes
Dec 4, 2018
TODO: Fix empty output case.
Add std::move().
Catch and translate locale creation exception to make the message meaningful.
yuslepukhin
force-pushed
the
dmitrism/string_normalizer
branch
from
73b5427 to
c549d83
Compare
RyanUnderhill
approved these changes
Dec 4, 2018
pranavsharma
pushed a commit
that referenced
this pull request
Dec 5, 2018
* Update Dockerfile for ARM build and build description * Update BUILD.md for ARM build * Fix typo * Kezhan/renaming graph_base.h to graph.h (#95) * rename graph.h to graph_viewer.h * rename graph_base.h to graph.h * Implement StringNormalizer (#69) * Imlpement StringNormalizer Add mixed language tests, test case insentive path. * Create a locale on the fly. Default locale does not seem to create well. * Add CI language-pack-en to make default locale available. Catch and translate locale creation exception to make the message meaningful. * Make sure locales are configured on Ubuntu. * Split the CI pipelines (#94) * Add the split build badges (#96) * Split the CI pipelines * Update build bdge display table in README * Revert Softmax optimizations using openmp. (#97) Revert "Simpler unused parameter in #if defined() switch." This reverts commit 6b00e6b. Revert "Better opemmp parallel group count calculation in Softmax parallel running." This reverts commit c530064. Revert "Optimize softmax cpu by parallel using openmp." This reverts commit e7bdfa0. * Delete logo as per marketing requirements (#98) * Update Dockerfile for ARM build and build description * Update BUILD.md for ARM build * Fix typo
TedThemistokleous
added a commit
to TedThemistokleous/onnxruntime
that referenced
this pull request
Jun 2, 2025
…soft#69) * Force hipify to use copied version from rocm-6.3.0-14776 build * Update hipify-perl location and permissions * Update hipify path to remove absolute path
quic-ankus
pushed a commit
to CodeLinaro/onnxruntime
that referenced
this pull request
Nov 25, 2025
Fuse scale into softmax
xiaofeihan1
added a commit
that referenced
this pull request
Jan 23, 2026
Some PRs that use core/common/inlined_containers.h can cause failures in the CUDA CI pipeline. ``` E:\_work\_temp\build\RelWithDebInfo\vcpkg_installed\x64-windows-static-md\include\absl/hash/internal/hash.h(481): error #68-D: integer conversion resulted in a change of sign [E:\_work\_temp\build\RelWithDebInfo\onnxruntime_providers_cuda.vcxproj] sizeof(T) == -1, ^ Remark: The warnings can be suppressed with "-diag-suppress <warning-number>" E:\_work\_temp\build\RelWithDebInfo\vcpkg_installed\x64-windows-static-md\include\absl/hash/hash.h(337): error #549-D: variable "s" is used before its value is set [E:\_work\_temp\build\RelWithDebInfo\onnxruntime_providers_cuda.vcxproj] return s; ^ E:\_work\_temp\build\RelWithDebInfo\vcpkg_installed\x64-windows-static-md\include\absl/container/internal/raw_hash_set.h(468): error #69-D: integer conversion resulted in truncation [E:\_work\_temp\build\RelWithDebInfo\onnxruntime_providers_cuda.vcxproj] static_cast<uint16_t>(reinterpret_cast<uintptr_t>(&seed)); ^ 3 errors detected in the compilation of "E:/_work/onnxruntime/onnxruntime/onnxruntime/contrib_ops/cuda/sparse/block_mask.cu". ``` This change adds a patch to Abseil to mitigate those failures. This solution has been verified to be effective in PR #27087.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Inputs:
X: tensor, string
Outputs:
Y: tensor, string
Attribute:
casechangeaction: enum {"LOWER", "UPPER", "NONE"}.
iscasesenstive: bool. Whether the identification of stop words in X is case-sensitive.
stopwords: list of strings (type: AttributeProto::STRINGS). A 1-D tensor of strings. Each element is a stop word.
locale: string. This attribute specifies culture for culture-specific string operations.
Description:
This operator can do below 2 operations:
[optional] Step1: Remove elements in X if they matches any of stop words so that output tensor may not contain any stop word. This operator only accepts [C]- and [1, C]-tensor. If all elements in X are dropped, the output will be the empty value of string tensor with shape [1] if input shape is [C] and shape [1, 1] if input shape is [1, C].
[optional] Step2: Lower all characters (if action is LOWER) in X or capitalize them (when action is UPPE